Papers with crowdsourcing tasks
TurkingBench: A Challenge Benchmark for Web Agents (2025.naacl-long)
Copied to clipboard
Kevin Xu, Yeganeh Kordi, Tanay Nayak, Adi Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, Daniel Khashabi
| Challenge: | TurkingBench is a benchmark consisting of tasks presented as web pages with textual instructions and multi-modal contexts. |
| Approach: | They propose to use HTML pages to perform various annotation tasks on crowdsourcing platforms. |
| Outcome: | The proposed model outperforms other models on the TurkingBench benchmark. |
JFCKB: Japanese Feature Change Knowledge Base (L18-1)
Copied to clipboard
| Challenge: | constructing commonsense knowledge including connotational meanings is challenging . a recent study focused on denotation and connotations, but few studies focused on connotating meanings . |
| Approach: | They propose to construct a Japanese knowledge base where arguments in event sentences are associated with feature changes caused by events. |
| Outcome: | The proposed knowledge base is able to generate anaphora resolution tasks in Japanese . it is useful for computers to understand texts, but it is difficult to acquire it . |
JDCFC: A Japanese Dialogue Corpus with Feature Changes (L18-1)
Copied to clipboard
| Challenge: | Existing corpora focus on emotional expressions in conversations, but there are no large-scale corpors focusing on the relationships between emotions and utterances. |
| Approach: | They propose a Japanese Feature Change Knowledge Base (JFCKB) that focuses on emotional expressions in conversations. |
| Outcome: | The proposed corpus can recognize reasonableness of a given conversation. |